Slides: https://www.dropbox.com/s/efdlptzl53hj3zi/wideio-actrec-tutorial-pydata-ldn2014.pdf

Source code: git clone https://bitbucket.org/wideio/pydata.git

• ~/Programming/pydata

Requires:

• Python
• NumPy / SciPy
• PIL
• OpenCV2

Why Python

• People usually come from MATLAB
• Now Python, a real and usable language, is much better
• Multiparadigm, operator overloading, lots of packages, community, ...

What is action recognition?

• Classification task.
• What are people doing in a video? Clapping, jogging, waving, boxing.
• KTH, Human Action Dataset

Supervised vs semi-supervised

• More priors: action specific, holistic approaches
• Less priors: feature-based, appearance-based, deep learning

Very traditional pipeline

Raw data -> Feature Extraction -> Machine Learning

• Insert domain-specific knowledge into parsing of raw data; hand-craft features

  Video Frames -> Bags of Keypoints -> SVM

Reading videos

• ffmpeg is the benchmark
• OpenCV ffmpeg wrapper is unmaintained though
• But if you also want sound probably want MLT (MLT Multimedia Framework)
  • Made by broadcasters
  • Backend of many open-source video editing software
  • brew install mlt
• Prefer pyglet to PIL
  • pyglet uses OpenGL, works on all platforms (not Windows 64)
  • or use QT.
  • (!!AI not a fan of PIL!)

Goal of this tutorial

Classify kissing and non-kissing pictures in video frames.

Harris Corner Detector

• Classic

Optical flow - wrapper based code

python keypoints/custom_feature.py

Feature extractions

• Image
• Produce a descriptor verctor using SIFT. Detect gradient.

See ~/Programming/pydata for more info. There are IPython Notebooks there too.

Process

Feature detction -> PCA Projections
-> Pick more important eigenvectors
-> Clustering
-> Graphic Model -> centers -> Bag of Words

• Feature detection stack for image
• PCA projection with graphic model
• Direction of projection to BOW cluster centers with metric
• Invert with 'discriminatpr' to turn into weighting
• Normalise (area = 1)
• Mean of stack to create histogram
• Then throw this result into the classifier

 How to run part3 BOW

cd ~/pydata/part3
ipython

>>> run bow.py
>>> demo()

# prints mean, centres, eigenvectors, but also saves model to file

>>> run featurecomputer.py
>>> demo()

# computes histogram for a particular file

>>> run svm_trainer_tester.py
>>> parambash()

# output param_bash_big.png. lighter colours are better.
# outputs a new direcory, opt3.1.2, with reports on classification